🕷️️ Job Radar • SCRAPING

upwork.com 🟢 2026-05-10

🔹 Scrape forum content and convert to a local AI knowledge base
👤 Client: 🇺🇸 United States Member since 2022-09-13
💰 Price: ****
🚩 Problem: Need to efficiently scrape a large forum for data and transform it into a usable format for an AI system.
📦 Existing: Not specified

Specifications:

[Target] Scrape approximately 11,000 pages from the forum
[Method] Use Python scraping tools like Scrapy, BeautifulSoup, Selenium, or Playwright
[UI/UX] Not applicable
[Stack] Python, Scrapy, BeautifulSoup, Selenium, Playwright, CSV, JSON, SQLite
[Security] Ensure responsible scraping without damaging the website; handle rate limits and proxies if needed
[Format] Deliver data in a usable format such as CSV, JSON, or SQLite

Workflow:

1. Analyze forum structure to determine best scraping approach.
2. Develop and test scraper using Python tools.
3. Scrape approximately 11,000 pages of the forum content.
4. Clean and organize scraped data by removing duplicates, spam, broken text, and irrelevant content.
5. Deliver cleaned dataset in a recommended format (CSV, JSON, SQLite).